Papers with prompt-based learning
MedTem2.0: Prompt-based Temporal Classification of Treatment Events from Discharge Summaries (2023.acl-srw)
Copied to clipboard
| Challenge: | Clinical texts contain important temporal information, such as medication start and end dates, appointment dates, and diagnosis dates. |
| Approach: | They propose to use prompt-based learning and fine-tuning to classify temporal relations between treatments and hospitalisation periods in discharge summaries. |
| Outcome: | The proposed method identifies whether a treatment was administered between the time of admission and discharge from the hospital. |
MEAL: Stable and Active Learning for Few-Shot Prompting (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for few-shot classification have high variance across different sets of few shots and finetuning runs. |
| Approach: | They propose novel ensembling methods that significantly reduce run variability and introduce a new active learning criterion for *data selection*. |
| Outcome: | The proposed method significantly reduces run variability and improves performance on five tasks. |
On Measuring Social Biases in Prompt-Based Multi-Task Learning (2022.findings-naacl)
Copied to clipboard
| Challenge: | a large body of work within prompt engineering attempts to understand the effects of input forms and prompts in achieving superior performance. |
| Approach: | They propose a large-scale text-to-text language model trained using prompts . they consider two different forms of semantically equivalent inputs - question-answer format and premise-hypothesis format . |
| Outcome: | The proposed model can generalize into novel forms of language and handle novel tasks. |
PromptExplainer: Explaining Language Models through Prompt-based Learning (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing explanation methods rely on linear approximations, accentuating irrelevant input tokens. |
| Approach: | They propose a method that aligns the explanation process with the masked language modeling task of pretrained language models and leverages prompt-based learning to generate class-dependent explanations. |
| Outcome: | Extensive experiments show that PromptExplainer outperforms state-of-the-art explanation methods. |
Distinguishability Calibration to In-Context Learning (2023.findings-eacl)
Copied to clipboard
| Challenge: | Recent studies have shown that pre-trained language models generate similar output embeddings which makes it difficult to discriminate for the prompt-based classifier. |
| Approach: | They propose a calibration method which rotates the embedding feature into a new metric space and adapts the ratio of each dimension to a uniform distribution. |
| Outcome: | The proposed method improves the distinguishability of learning embeddings on three datasets under various settings. |
SparseFit: Few-shot Prompting with Sparse Fine-tuning for Jointly Generating Predictions and Natural Language Explanations (2024.acl-long)
Copied to clipboard
| Challenge: | Models that generate natural language explanations (NLEs) for their predictions often require large datasets of human-written NLEs at training time, which can be expensive and time-consuming to collect. |
| Approach: | They propose a sparse few-shot fine-tuning strategy that leverages discrete prompts to jointly generate predictions and NLEs. |
| Outcome: | The proposed approach compares sparse few-shot fine-tuning with existing parametric fine- tuning techniques on three sizes of the T5 language model and four datasets and produces competitive results for both task performance and NLE quality. |
Cold-Start Data Selection for Better Few-shot Language Model Fine-tuning: A Prompt-based Uncertainty Propagation Approach (2023.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) have achieved competitive performance with limited labeled data for many NLP tasks. |
| Approach: | They propose a prompt-based data selection method for pre-trained language models fine-tuning under cold-start scenarios. |
| Outcome: | The proposed method outperforms the strongest cold-start data selection baselines on six text classification datasets with 128 labels. |
Hierarchical Verbalizer for Few-Shot Hierarchical Text Classification (2023.acl-long)
Copied to clipboard
| Challenge: | Existing work on the hierarchical text classification problem is limited due to the complexity of label hierarchy and intensive labeling cost. |
| Approach: | They propose a path-based few-shot setting and a strict path-basic evaluation metric to further explore few- shot HTC tasks. |
| Outcome: | The proposed framework outperforms those who inject hierarchy through graph encoders on three popular HTC datasets under the few-shot setting. |
Good Examples Make A Faster Learner: Simple Demonstration-based Learning for Low-resource NER (2022.acl-long)
Copied to clipboard
Dong-Ho Lee, Akshen Kadakia, Kangmin Tan, Mahak Agarwal, Xinyu Feng, Takashi Shibuya, Ryosuke Mitani, Toshiyuki Sekiya, Jay Pujara, Xiang Ren
| Challenge: | Recent advances in prompt-based learning have shown strong results on few-shot text classification by using cloze-style templates. |
| Approach: | They propose a demonstration-based learning method which lets the input be prefaced by task demonstrations for in-context learning. |
| Outcome: | The proposed method improves on in-domain learning and domain adaptation in low-resource settings. |
A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Models (2022.acl-long)
Copied to clipboard
| Challenge: | Recent few-shot learning models such as GPT3 are expensive and slow to deploy for real-world applications. |
| Approach: | They propose a prompt-based low-resource learning method for VL tasks with a few examples . they pre-train a sequence-to-sequence transformer model with prefix and masked language modeling . |
| Outcome: | The proposed method outperforms Frozen on vision-language tasks with prompt-based learning by 18.2% point. |
DESED: Dialogue-based Explanation for Sentence-level Event Detection (2022.coling-1)
Copied to clipboard
Yinyi Wei, Shuaipeng Liu, Jianwei Lv, Xiangyu Xi, Hailei Yan, Wei Ye, Tong Mo, Fan Yang, Guanglu Wan
| Challenge: | Existing methods for sentence-level event detection depend on manual annotations or domain expertise to design sophisticated templates and rules. |
| Approach: | They propose a dialogue-based explanation paradigm to enhance sentence semantics for event detection. |
| Outcome: | The proposed method can be applied to two event detection datasets. |
Learning to Transfer Prompts for Text Generation (2022.naacl-main)
Copied to clipboard
| Challenge: | Pretrained language models (PLMs) have made remarkable progress in text generation tasks via fine-tuning. |
| Approach: | They propose a prompt-based method that learns source prompts and transfers them as target prompts to perform target generation tasks. |
| Outcome: | The proposed method can be used to perform text generation tasks in a transferable setting. |
What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers (2021.emnlp-main)
Copied to clipboard
Boseop Kim, HyoungSeok Kim, Sang-Woo Lee, Gichang Lee, Donghyun Kwak, Jeon Dong Hyeon, Sunghyun Park, Sungju Kim, Seonhoon Kim, Dongpil Seo, Heungsub Lee, Minyoung Jeong, Sungjae Lee, Minsub Kim, Suk Hyun Ko, Seokhun Kim, Taeyong Park, Jinuk Kim, Soyoung Kang, Na-Hyeon Ryu, Kang Min Yoo, Minsuk Chang, Soobin Suh, Sookyo In, Jinseong Park, Kyungduk Kim, Hiun Kim, Jisu Jeong, Yong Goo Yeo, Donghoon Ham, Dongju Park, Min Young Lee, Jaewook Kang, Inho Kang, Jung-Woo Ha, Woomyoung Park, Nako Sung
| Challenge: | GPT-3 has been used to train large-scale language models on hundreds of billion scale data. |
| Approach: | They propose a Korean variant of GPT-3 that uses Korean tokens to train in-context models. |
| Outcome: | The proposed method shows state-of-the-art zero-shot and few-shot learning on downstream tasks in Korean. |
NSP-BERT: A Prompt-based Few-Shot Learner through an Original Pre-training Task —— Next Sentence Prediction (2022.coling-1)
Copied to clipboard
| Challenge: | Recent studies have shown that using prompts to utilize language models to perform downstream tasks is more effective than using token-level methods such as PET. |
| Approach: | They propose to use a BERT original pre-training task abandoned by RoBERTa and other models to construct a sentence-level prompt-based method that does not need to fix the length of the prompt or the position to be predicted. |
| Outcome: | The proposed method performs better than PET and EFL on a BERT pre-training task and is comparable to other prompt-based methods. |
LinkPrompt: Natural and Universal Adversarial Attacks on Prompt-based Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Prompt-based learning is a new language model training paradigm that adapts Pre-trained Language Models (PLMs) to downstream tasks. |
| Approach: | They propose a prompt-based learning paradigm that adapts Pre-trained Language Models to downstream tasks . they use a gradient-based beam search algorithm to generate adversarial triggers . |
| Outcome: | The proposed model improves performance on various natural language processing tasks by optimizing the prompt template. |
HQP: A Human-Annotated Dataset for Detecting Online Propaganda (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing datasets for detecting online propaganda use weak labels that can be noisy and incorrect. |
| Approach: | They propose a dataset for detecting online propaganda with high-quality labels . they show that state-of-the-art language models fail in detecting propaganda when trained with weak labels compared to prompt-based learning . |
| Outcome: | The proposed dataset is the first large-scale dataset for detecting online propaganda that was created through human annotation. |
Decorate the Examples: A Simple Method of Prompt Design for Biomedical Relation Extraction (2022.lrec-1)
Copied to clipboard
| Challenge: | Recent research shows that prompt-based learning improves performance on relation extraction tasks. |
| Approach: | They propose a prompt-based learning method that generates comprehensive prompts for biomedical relation extraction using a ChemProt dataset. |
| Outcome: | The proposed method improves fine-tuning on a biomedical relation extraction task with a cloze-test task and fewer training examples to make reasonable predictions. |
TabPrompt: Graph-based Pre-training and Prompting for Few-shot Table Understanding (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods of Table Understanding (TU) focus on the textual content within the tabular data, disregarding the topological information of the table. |
| Approach: | They propose a framework that uses tabs to understand tabular data without ignoring the topological information of the table. |
| Outcome: | The proposed framework outperforms baselines in few-shot table understanding tasks. |
Argument Mining in Data Scarce Settings: Cross-lingual Transfer and Few-shot Techniques (2024.acl-long)
Copied to clipboard
| Challenge: | Recent work on sequence labelling has explored different strategies to mitigate the lack of manually annotated data for the large majority of the world languages. |
| Approach: | They propose to use the mask objective to exploit the few-shot capabilities of pre-trained language models to improve their performance. |
| Outcome: | The proposed model-transfer outperforms data-transference and fine-tuning outperformed few-shot methods for Argument Mining task. |
Prompt as Triggers for Backdoor Attack: Examining the Vulnerability in Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | ProAttack is a novel and efficient method for performing clean-label backdoor attacks based on the prompt, which uses the prompt itself as a trigger. |
| Approach: | They propose a method for performing clean-label backdoor attacks based on the prompt, which uses the prompt itself as a trigger. |
| Outcome: | The proposed method achieves state-of-the-art performance on several NLP tasks, particularly in few-shot settings. |
Adversarial Knowledge Stimulated Contrastive Prompting for Few-shot Language Learners (2023.findings-acl)
Copied to clipboard
| Challenge: | Prompt-based fine-tuning has boosted performance of Pre-trained language models on few-shot Natural Language Understanding (NLU) tasks by employing task-specific prompts. |
| Approach: | They propose a Cloze-driven prompt framework for prompt tuning that implicitly stimulates knowledge from pre-trained language models. |
| Outcome: | The proposed framework outperforms state-of-the-art for prompt-based fine-tuning on few-shot NLU tasks. |